Improve HIP docs on fat binary registration ordering #168566
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
@llvm/pr-subscribers-clang Author: Yaxun (Sam) Liu (yxsamliu) ChangesClarify how Clang-generated HIP fat binaries are registered and unregistered with the HIP runtime, and how this interacts with global constructors, destructors, and atexit handlers. Document that there is no strong guarantee on ordering relative to user-defined global ctors/dtors, recommend that HIP application developers avoid using kernels or device variables from global ctors/dtors, and describe the implications for HIP runtime developers (synchronization and guards in __hipRegisterFatBinary/__hipUnregisterFatBinary). This is motivated by questions from HIP application and runtime developers about fat binary registration/unregistration order and its potential interference with their own initialization and teardown code. Full diff: https://github.com/llvm/llvm-project/pull/168566.diff 1 Files Affected:
diff --git a/clang/docs/HIPSupport.rst b/clang/docs/HIPSupport.rst
index ab9ea110e6d54..b33d663f0cfee 100644
--- a/clang/docs/HIPSupport.rst
+++ b/clang/docs/HIPSupport.rst
@@ -210,6 +210,88 @@ Host Code Compilation
- These relocatable objects are then linked together.
- Host code within a TU can call host functions and launch kernels from another TU.
+HIP Fat Binary Registration and Unregistration
+=============================================
+
+When compiling HIP for AMD GPUs, Clang embeds device code into HIP "fat
+binaries" and generates host-side helper functions that register these
+fat binaries with the HIP runtime at program start and unregister them at
+program exit. In non-RDC mode (``-fno-gpu-rdc``), each compilation unit
+typically produces its own self-contained fat binary per GPU architecture. In
+RDC mode (``-fgpu-rdc``), device bitcode from multiple compilation units may be
+linked together into a single fat binary per GPU architecture.
+
+At the LLVM IR level, Clang/LLVM typically create an internal module
+constructor (for example ``__hip_module_ctor`` or a ``.hip.fatbin_reg``
+function) and add it to ``@llvm.global_ctors``. This constructor is called by
+the C runtime before ``main`` and it:
+
+* calls ``__hipRegisterFatBinary`` with a pointer to an internal wrapper
+ object that describes the HIP fat binary;
+* stores the returned handle in an internal global variable;
+* calls an internal helper such as ``__hip_register_globals`` to register
+ kernels, device variables and other metadata associated with the fat binary;
+* registers a corresponding module destructor with ``atexit`` so it will run
+ during program termination.
+
+The module destructor (for example ``__hip_module_dtor`` or a
+``.hip.fatbin_unreg`` function) loads the stored handle, checks that it is
+non-null, calls ``__hipUnregisterFatBinary`` to unregister the fat binary from
+the HIP runtime, and then clears the handle. This ensures that the HIP runtime
+sees each fat binary registered exactly once and that it is unregistered once
+at exit, even when multiple translation units contribute HIP kernels to the
+same host program.
+
+These registration/unregistration helpers are implementation details of Clang's
+HIP code generation; user code should not call ``__hipRegisterFatBinary`` or
+``__hipUnregisterFatBinary`` directly.
+
+Implications for HIP Application Developers
+------------------------------------------
+
+The fat binary registration and unregistration helpers participate in the same
+global constructor and termination mechanisms as the rest of the program, and
+there is no strong guarantee about their relative order with user-defined
+global constructors and destructors. In particular:
+
+* Applications should not invoke ``__hipRegisterFatBinary`` or
+ ``__hipUnregisterFatBinary`` explicitly.
+* Because registration happens in a compiler-generated module constructor and
+ unregistration happens via an ``atexit``-registered module destructor, the
+ exact ordering relative to other global ctors/dtors and ``atexit`` handlers
+ is implementation-dependent and may vary across platforms and toolchain
+ options.
+* To avoid subtle ordering issues, applications should not rely on HIP kernels
+ or device variables being usable from user-defined global constructors or
+ destructors. HIP initialization and teardown that touches kernels or device
+ state should instead be performed in ``main`` (or in functions called from
+ ``main``) after process startup.
+* In RDC mode, multiple translation units may contribute device code to a
+ single fat binary; user code should not make assumptions based on a
+ particular registration order between translation units.
+
+Implications for HIP Runtime Developers
+--------------------------------------
+
+HIP runtime implementations that are linked with Clang-generated host code
+must handle registration and unregistration in the presence of uncertain
+global ctor/dtor ordering:
+
+* ``__hipRegisterFatBinary`` must accept a pointer to the compiler-generated
+ wrapper object and return an opaque handle that remains valid for as long as
+ the fat binary may be used.
+* ``__hipUnregisterFatBinary`` must accept the handle previously returned by
+ ``__hipRegisterFatBinary`` and perform any necessary cleanup. It may be
+ called late in process teardown, after other parts of the runtime have
+ started shutting down, so it should be robust in the presence of partially
+ torn-down state.
+* Runtimes should use appropriate synchronization and guards so that fat
+ binary registration does not observe uninitialized resources and
+ unregistration does not release resources that are still required by other
+ runtime components. In particular, registration and unregistration routines
+ should be written to be safe under repeated calls and in the presence of
+ concurrent or overlapping initialization/teardown logic.
+
Syntax Difference with CUDA
===========================
|
yxsamliu
force-pushed
the
fatbin-reg-doc
branch
from
e0dc0df to
f007972
Compare
Artem-B
approved these changes
Nov 18, 2025
Member
Artem-B
left a comment
Artem-B
left a comment
There was a problem hiding this comment.
LGTM overall, with a few editorial nits.
yxsamliu
force-pushed
the
fatbin-reg-doc
branch
from
f007972 to
3f1e940
Compare
Clarify how Clang-generated HIP fat binaries are registered and unregistered with the HIP runtime, and how this interacts with global constructors, destructors, and atexit handlers. Document that there is no strong guarantee on ordering relative to user-defined global ctors/dtors, recommend that HIP application developers avoid using kernels or device variables from global ctors/dtors, and describe the implications for HIP runtime developers (synchronization and guards in __hipRegisterFatBinary/__hipUnregisterFatBinary). This is motivated by questions from HIP application and runtime developers about fat binary registration/unregistration order and its potential interference with their own initialization and teardown code.
yxsamliu
force-pushed
the
fatbin-reg-doc
branch
from
3f1e940 to
0ab0805
Compare
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/66/builds/22772 Here is the relevant piece of the build log for the reference |
aadeshps-mcw
pushed a commit
to aadeshps-mcw/llvm-project
that referenced
this pull request
Nov 26, 2025
Clarify how Clang-generated HIP fat binaries are registered and unregistered with the HIP runtime, and how this interacts with global constructors, destructors, and atexit handlers. Document that there is no strong guarantee on ordering relative to user-defined global ctors/dtors, recommend that HIP application developers avoid using kernels or device variables from global ctors/dtors, and describe the implications for HIP runtime developers (synchronization and guards in __hipRegisterFatBinary/__hipUnregisterFatBinary). This is motivated by questions from HIP application and runtime developers about fat binary registration/unregistration order and its potential interference with their own initialization and teardown code.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Clarify how Clang-generated HIP fat binaries are registered and unregistered with the HIP runtime, and how this interacts with global constructors, destructors, and atexit handlers. Document that there is no strong guarantee on ordering relative to user-defined global ctors/dtors, recommend that HIP application developers avoid using kernels or device variables from global ctors/dtors, and describe the implications for HIP runtime developers (synchronization and guards in __hipRegisterFatBinary/__hipUnregisterFatBinary). This is motivated by questions from HIP application and runtime developers about fat binary registration/unregistration order and its potential interference with their own initialization and teardown code.